Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network

We analytically derive the geometrical structure of the weight space in multilayer neural networks (MLN), in terms of the volumes of couplings associated to the internal representations of the training set. Focusing on the parity and committee machines, we deduce their learning and generalization capabilities both reinterpreting some known properties and finding new exact results. The relationship between our approach and information theory as well as the Mitchison--Durbin calculation is established. Our results are exact in the limit of a large number of hidden units, showing that MLN are a class of exactly solvable models with a simple interpretation of replica symmetry breaking.Comment: 12 pages, 1 compressed ps figure (uufile), RevTeX fil

Relationship between clustering and algorithmic phase transitions in the random k-XORSAT model and its NP-complete extensions

We study the performances of stochastic heuristic search algorithms on Uniquely Extendible Constraint Satisfaction Problems with random inputs. We show that, for any heuristic preserving the Poissonian nature of the underlying instance, the (heuristic-dependent) largest ratio $\alpha_a$ of constraints per variables for which a search algorithm is likely to find solutions is smaller than the critical ratio $\alpha_d$ above which solutions are clustered and highly correlated. In addition we show that the clustering ratio can be reached when the number k of variables per constraints goes to infinity by the so-called Generalized Unit Clause heuristic.Comment: 15 pages, 4 figures, Proceedings of the International Workshop on Statistical-Mechanical Informatics, September 16-19, 2007, Kyoto, Japan; some imprecisions in the previous version have been correcte

Slow nucleic acid unzipping kinetics from sequence-defined barriers

Recent experiments on unzipping of RNA helix-loop structures by force have shown that about 40-base molecules can undergo kinetic transitions between two well-defined `open' and `closed' states, on a timescale = 1 sec [Liphardt et al., Science 297, 733-737 (2001)]. Using a simple dynamical model, we show that these phenomena result from the slow kinetics of crossing large free energy barriers which separate the open and closed conformations. The dependence of barriers on sequence along the helix, and on the size of the loop(s) is analyzed. Some DNAs and RNAs sequences that could show dynamics on different time scales, or three(or more)-state unzipping, are proposed.Comment: 8 pages Revtex, including 4 figure

Trajectories in phase diagrams, growth processes and computational complexity: how search algorithms solve the 3-Satisfiability problem

Most decision and optimization problems encountered in practice fall into one of two categories with respect to any particular solving method or algorithm: either the problem is solved quickly (easy) or else demands an impractically long computational effort (hard). Recent investigations on model classes of problems have shown that some global parameters, such as the ratio between the constraints to be satisfied and the adjustable variables, are good predictors of problem hardness and, moreover, have an effect analogous to thermodynamical parameters, e.g. temperature, in predicting phases in condensed matter physics [Monasson et al., Nature 400 (1999) 133-137]. Here we show that changes in the values of such parameters can be tracked during a run of the algorithm defining a trajectory through the parameter space. Focusing on 3-Satisfiability, a recognized representative of hard problems, we analyze trajectories generated by search algorithms using growth processes statistical physics. These trajectories can cross well defined phases, corresponding to domains of easy or hard instances, and allow to successfully predict the times of resolution.Comment: Revtex file + 4 eps figure

Theory of spike timing based neural classifiers

We study the computational capacity of a model neuron, the Tempotron, which classifies sequences of spikes by linear-threshold operations. We use statistical mechanics and extreme value theory to derive the capacity of the system in random classification tasks. In contrast to its static analog, the Perceptron, the Tempotron's solutions space consists of a large number of small clusters of weight vectors. The capacity of the system per synapse is finite in the large size limit and weakly diverges with the stimulus duration relative to the membrane and synaptic time constants.Comment: 4 page, 4 figures, Accepted to Physical Review Letters on 19th Oct. 201

Tricritical Points in Random Combinatorics: the (2+p)-SAT case

The (2+p)-Satisfiability (SAT) problem interpolates between different classes of complexity theory and is believed to be of basic interest in understanding the onset of typical case complexity in random combinatorics. In this paper, a tricritical point in the phase diagram of the random $2+p$-SAT problem is analytically computed using the replica approach and found to lie in the range $2/5 \le p_0 \le 0.416$. These bounds on $p_0$ are in agreement with previous numerical simulations and rigorous results.Comment: 7 pages, 1 figure, RevTeX, to appear in J.Phys.